Almost all computer systems currently in existence implement a cache memory system. A cache memory is typically a random access memory (RAM) that a computer microprocessor accesses more quickly than it can access main memory. As the microprocessor processes data, it looks first in the cache, and if it finds the data in the cache, the microprocessor does not have to do the more time-consuming reading of data from larger main memory.
Typically, computer systems implement a conventional cache organization that implements caches L1 through L2 or L3. When processing data, the microprocessor first looks in the L1 cache for the data. If the data is not found in the L1 cache, the L2 cache is searched for the data. Finally the L3 cache is searched if the data is not found in the L2 cache.
A problem with the conventional cache organization is that data associated with various instructions are more critical than others, and thus need to be processed faster (e.g., 1 latency cycle). However, conventional cache operation does not account for the fast retrieval of such critical data from the cache system. For example, both critical data and non-critical data (e.g., data requiring up to 4 latency cycles) may be stored in the L1 cache. Storing non-critical data at the fast L1 cache is inefficient because it reduces the number of critical loads data that may be stored in the L1 cache. Thus, the larger, slower L2 or L3 caches must be searched to retrieve critical data that will are not stored within the L1 cache, exceeding the 1 latency cycle.